Systems and methods for generating accurate transaction data and manipulation

ABSTRACT

In a system and method for auditing transactions where a set of image based transactions are received over a communications network, and stored in a central data store, the set of image based transactions are associated with a unique identifier associated with a user. A transaction format is identified from the set of image based transactions, utilizing a processor to apply a preprocessing to the set of image based transaction based on the identifying. The preprocessed image based transactions are processed into a series of text based transactions, wherein each image based transaction has a related text based transactions and each text based transaction has a plurality of data representing the transaction. The plurality of data for each text based transaction is stored and a quality identifier is associated with each text based transaction. An identifier is applied to a text based transaction based on the quality identifier.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a Divisional of U.S. Nonprovisionalapplication Ser. No. 15/420,005 filed 30 Jan. 2017; which claims thebenefit of U.S. Provisional Application Ser. 62/288,627 filed 29 Jan.2016; each of which is incorporated herein by reference in its entiretyfor all purposes.

SUMMARY

In general, the present disclosure is directed at producing highlyaccurate transaction histories for a series of transaction input over anetwork, and providing efficient verification and error correction. Morespecifically, the present system and methods disclosed are directedtoward systems and methods for accurately inputting a series offinancial transaction from a statement quickly, with a high degree ofaccuracy and a reduction in processing time and then compiling the datatransactions for manipulation.

BACKGROUND

Many people and entities continue to receive bank records, credit cardstatements and other financial statements in traditional paper format orelectronic images resembling traditional paper format. Whether receivingthese transaction records in paper of Portable Document File (PDF)format reviewing, reconciling and organizing and quickly searching isdifficult or impossible because transactions span across differentstatements, may be from different sources, and are impossible tomanually reorganize or regroup in their original physical form.

If a user wanted to enter the data into a tabulated format, such as aspreadsheet, errors are easily introduced. Accordingly a need exists fora system and method that ensures near one-hundred percent accuracy indata input, allows for manipulation, filtering, grouping and searching.

SUMMARY

The current system and method allows users to scan and upload bankstatements, credit card statements, investment statements, and any othertabulated series of repeated entries into a web based application forprocessing and to identify errors in the entries, sort search and filterthe entries. The system and methods utilizing preprocessing techniquesimprove processing times, increases accuracy and reduces errors.

In one or more embodiments, the uploaded statement images arepreprocessed and converted to text using Optical Character Recognition(OCR) to identify transactions and transaction types. The transactionsare recognized, organized, extracted and tabulated into a database andthen may be displayed on a display using a graphical user interface. Inan embodiment, the processing occurs over a network. Conversion ofuploaded documents can be done quickly and efficiently in a matter ofminutes by employing multiple core processors operating in parallel inaccordance with embodiments of the present disclosure.

In one or more embodiments, before the user begins searching and sortingthrough data strings and transactions in the database, transactionaldata are verified via an error correction procedure that ensuresone-hundred percent accuracy. In an embodiment, the system and methodutilize the extracted statement account balances to autonomouslyidentify potential OCR conversion mistakes.

In one or more embodiments, if the system fails to read a transactionvalue correctly, the user is prompted to that failure via highlightingof the failure on the display. In another embodiment, highlighting mayindicate that the data was read but that error correction detected someproblem with the conversion. In an embodiment, potential reading errorsare displayed on the GUI alongside or in conjunction with the originalsource data, thereby allowing the user to view the original andconverted data simultaneously. In an embodiment, when a user revises apotential error, the remaining data entries, which might have beenpreviously identified as potential errors are recomputed and reevaluatedautomatically based on the revision. Upon completion of the review, thesystem compiles a one-hundred percent accurate data store of transactionprocessing.

In one or more embodiments, the user interfacing with the system througha GUI, may sort, search and filter through the transactions data storeto locate transactions of interest. In an embodiment, the searches maybe based on date, dollar amount and description. In an embodiment, eachtransaction in the data store is linked back to the original location onthe original document image. In an embodiment, when a transaction in thedatabase is selected, a snapshot image of the appropriate page from theuploaded statement is displayed, with the corresponding transactionhighlighted. Accordingly, a user can quickly contextualize thetransaction to determine its significance to the audit.

In one or more embodiments, a user audits a transaction by receiving,over a communications network, a set of image based transactions,storing the set of image based transaction in a central data store,associating the set of image based transactions with a unique identifierassociated with a user, identifying a transaction format from the set ofimage based transactions, utilizing a processor to apply a preprocessingto the set of image based transactions based on the identifying,processing the preprocessed image based transactions, via the processor,into a series of text based transactions, wherein each image basedtransaction has a related text based transaction and each text basedtransaction has a plurality of data representing the transaction,storing, the plurality of data for each text based transaction in a datastore, associating a quality identifier with each text based transactionwherein the quality identifier is based on the probability that the textbased transaction was processed correctly, applying an identifier to atext based transaction based on the quality identifier, andtransmitting, over the network, to a display the text based transactionand the associated identifier.

In one or more embodiments, conversion of an image of a document totextual data is improved by using a microprocessor for preprocessing thedocument image, applying optical character recognition to the documentimage, using the microprocessor, to convert the document image into aset of textual data, identifying a first subset of data comprising oneor more items from the set of textual data, using the microprocessor,based on a similarity between the one or more items to each other or toa predefined template, annotating the first subset of data, using themicroprocessor, analyzing the annotated first subset of data to identifyeach of the one or more items as a unique transaction, associating afirst unique transaction with a key identifier, analyzing at least oneof the unique transactions other than the first unique transaction basedon the key identifier to determine a correlation between the firstunique transaction and the at least one of the unique transactions otherthan the first unique transaction, and determining, based on thecorrelation, whether the subset of data has been accurately converted totextual data.

In one or more embodiments, the preprocessing includes at least one ofde-skewing, de-specking, binarization, line removal, layout analysis,zoning, line detection, word detection, character isolation,segmentation, applying a lexicon, near neighbor analysis, grammaticalanalysis, applying Benfords law, or any combinations thereof.

In one or more embodiments, the preprocessing identifies transactionsand transaction types. In other embodiments, each unique transaction isat least one of recognized, identified, organized, extracted, ortabulated into a database. In further embodiments, the document is abank statement or a credit card statement. In additional embodiments,the key identifier is at least one of a running balance, identifiedtotal, account number, or date range.

In one or more embodiments, the extracted bank statement or credit cardstatement account balances are used to autonomously identify potentialOCR conversion errors. In other embodiments, potential errors arepresented on a graphical user interface in conjunction with the originalportion of the document image corresponding to the first subset of datafor comparison purposes.

In one or more embodiments, the set of textual data is stored in a datastore, wherein each unique transaction comprising the set of textualdata in the data store is linked back to the original location on thedocument image. In other embodiments, questionable unique transactionsare flagged. In further embodiments, the key identifier is used toidentify and flag potential OCR conversion errors.

In one or more embodiments, textual data is edited to provide foraccurate conversion. In other embodiments, upon editing of at least oneportion of textual data identified as a potential error, the remainingportions of textual data, which may have been identified as potentialerrors are automatically reevaluated based on the edits.

In one or more embodiments, the annotating includes at least one ofhighlighting unique transactions verified as accurate, highlightingunique transactions identified as containing errors, or highlightingunique transactions verified as accurate and unique transactionsidentified as containing errors. In other embodiments, the uniquetransactions verified as accurate are highlighted in a first color andthe unique transactions identified as containing errors are highlightedin a second color.

In one or more embodiments, the data store is searched for one or moreunique transactions. In other embodiments, the graphical user interfaceincludes a user created button for saving an advanced search parameterfor searching the data store. In further embodiments, the graphical userinterface includes a slider button to set a closeness value for a searchparameter for searching the data store. In additional embodiments, thefirst unique transaction is set as an anchor transaction to use as basisfor an advanced search. In yet other embodiments, a recurringtransaction is identified using the anchor transaction as the basis forthe search.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary depiction of the user upload screen prompting auser to upload scanned or electronic statements in an embodiment of thepresent disclosure.

FIG. 2 is an exemplary depiction of user analyze screen prompting a userto view and search through converted transactions in an embodiment ofthe present disclosure.

FIG. 3 is an exemplary depiction of a series of transactions in anembodiment of the present disclosure.

FIG. 4 is an exemplary depiction of a series of transactions in anembodiment indicating that some of the transactions were not convertedto text from the original documents in an embodiment of the presentdisclosure.

FIG. 5 is an exemplary depiction of the verification mode in anembodiment of the present disclosure.

FIG. 6 is an exemplary depiction of a transaction that was not enteredresulting in an omitted entry in an embodiment of the presentdisclosure.

FIG. 7 is an exemplary depiction of a newly added/previously omittedtransaction being entered in an embodiment of the present disclosure.

FIG. 8 is an exemplary depiction of a revised transaction data storeafter a previously omitted transaction is entered in an embodiment ofthe present disclosure.

FIG. 9 is an exemplary depiction of the text beneath a check image beingmisidentified as a transaction in an embodiment of the presentdisclosure.

FIG. 10 is an exemplary depiction of the investigation modes screenafter all the data verification has been completed in an embodiment ofthe present disclosure.

FIG. 11 is an exemplary depiction of a search screen for a searchseeking all deposit transactions in an embodiment of the presentdisclosure.

FIG. 12 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded in an embodiment of the present disclosure.

FIG. 13 is an exemplary depiction of a search screen showing the abilityto search data aggregated across different account in an embodiment ofthe present disclosure.

FIG. 14 is an exemplary depiction of a screen showing all transfers andall exact dollar amount transfers between accounts in an embodiment ofthe present disclosure.

FIG. 15 is an exemplary depiction of a screen showing the matchingtransaction feature in an embodiment of the present disclosure.

FIG. 16 is an exemplary depiction of a screen showing a parameterselection screen in an embodiment of the present disclosure.

FIG. 17 is an exemplary depiction of a screen showing an anchortransaction in an embodiment of the present disclosure.

FIG. 18 is an exemplary depiction of screen showing a parameterselection screen in an embodiment of the present disclosure.

FIG. 19 is an exemplary depiction of a screen showing the results of atypical search in an embodiment of the present disclosure.

FIG. 20 is an exemplary depiction of a screen showing a saved search inan embodiment of the present disclosure.

FIG. 21 is an illustration showing an exemplary system in accordancewith an embodiment of the present disclosure.

FIG. 22 is an illustration showing the components in an exemplary systemin accordance with an embodiment of the present disclosure.

FIG. 23 is an exemplary chart of a scanned document during the OCRprocess in accordance with an embodiment of the present disclosure.

FIG. 24 is an example of a Directed Acyclic Graph in accordance with anembodiment of the present disclosure.

FIG. 25 is an example of a GPU memory map in accordance with anembodiment of the present disclosure.

FIG. 26 is an example of the work flow of an exemplary machine learningengine in accordance with an embodiment of the present disclosure

FIG. 27 is a schematic of an exemplary computing system for implementingthe system in accordance with an embodiment of the present disclosure.

FIG. 28 is an example of a workflow for a portion of a review processaccording to an embodiment of the present disclosure.

FIG. 29 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded showing a highlighted portion according to anembodiment of the present disclosure.

FIG. 30 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded showing annotated portions according to anembodiment of the present disclosure.

FIG. 31 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded showing a data entry portion for correcting dataaccording to an embodiment of the present disclosure.

FIG. 32 is an example of a workflow for a portion of a review processaccording to an embodiment of the present disclosure.

FIG. 33 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded showing a highlighted portion and a data entryportion for correcting data according to an embodiment of the presentdisclosure.

FIG. 34 is an example of a workflow for a portion of a review processaccording to an embodiment of the present disclosure.

FIG. 35 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded showing a data entry portion and an annotatedportion for correcting data according to an embodiment of the presentdisclosure.

FIG. 36 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded showing a data entry portion for correcting dataaccording to an embodiment of the present disclosure.

FIG. 37 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded showing a data entry portion for correcting dataaccording to an embodiment of the present disclosure.

FIG. 38 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded showing a highlighted portion for reconciliationof data according to an embodiment of the present disclosure.

FIG. 39 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded showing a highlighted portion for reconciliationof data according to an embodiment of the present disclosure.

FIG. 40 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded showing a highlighted portion for reconciliationof data according to an embodiment of the present disclosure.

FIG. 41 is an exemplary depiction of a “snapshot” image of a page fromthe original uploaded showing a portion for reconciliation of dataaccording to an embodiment of the present disclosure.

FIG. 42 is an example of a workflow for a review process according to anembodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure is now described more fully with reference to theaccompanying drawings, in which example embodiments of the presentdisclosure are shown. The present disclosure may, however, be embodiedin many different forms and should not be construed as necessarily beinglimited to the example embodiments disclosed herein. Rather, theseexample embodiments are provided so that the present disclosure isthorough and complete, and fully conveys the concepts of the presentdisclosure to those skilled in the relevant art. In addition, featuresdescribed with respect to certain example embodiments may be combined inand/or with various other example embodiments. Different aspects and/orelements of example embodiments, as disclosed herein, may be combined ina similar manner. Further, some example embodiments, whetherindividually and/or collectively, may be components of a larger system,wherein other procedures may take precedence over and/or otherwisemodify their application. Additionally, a number of steps may berequired before, after, and/or concurrently with example embodiments, asdisclosed herein. Note that any and/or all methods and/or processes, atleast as disclosed herein, can be at least partially performed via atleast one entity in any manner. The current system uploads documents inscanned or electronic formats for searching, processing, and sorting inan error free environment. In an embodiment, the uploaded documents werefinancial documents, such as bank statements, investment accountstatements, credit card statements or any other type of statementreflecting a series of repeated or repeating transactions.

The documents may be uploaded over a network. The network may be asingle network or a combination of different networks. For example, anetwork can be a local area network (LAN), a wide area network (WAN), apublic network, a private network, a proprietary network, a PublicTelephone Switched Network (PSTN), the Internet, a wireless network, avirtual network, or any combination thereof. A network may also includevarious network access points, e.g., wired or wireless access pointssuch as base stations or Internet exchange points, through which a usermay connect to the network in order to receive and process data.

FIG. 1 depicts a data entry screen prompting the user to uploaddocuments over a network. As will be understood, this may be the firsttime a user uploads documents or may be a subsequent upload to anexisting account. It may be a new individual document or a series ofdocuments. Documents may be uploaded in one of several formats includingPDF, PCL, Postscript, TIFF, GIF, JPEG, BMP, PNG, or any other fileformat comprising text, objects, and images.

Once uploaded, the images are converted to text and the text isextracted using Optical Character Recognition (OCR). The identifiedtransactions are recognized, extracted and tabulated in a data store ordatabase, which can then be accessed by the system. Text extraction maybe done in minutes based on the allocated resources. Before a user ofthe system can efficiently utilize, search and sort through the datastore of transactions, the data may need to be verified utilizing anerror correction procedure, that ensures one-hundred percent accuracy.The present systems and methods use key identifiers to identify and flagpotential OCR conversion errors. For example, when dealing withfinancial transaction, running balances or identified totals may be usedas a key identifier of errors in the data in the data store.

FIG. 2 illustrates a screen of the user interface prompting the user toanalyze the uploaded transactions for errors. By selecting this step,the user will be directed toward potential OCR conversion errors thatwere detected and identified automatically by the system during documentconversion and data extraction.

In an embodiment, the account balances may be used to automatically andautonomously identify potential OCR conversion errors. For example, ifthe most recent balance in a transaction was $2,300.00 and the nextbalance is $2,425.00, the present system can identify that the value ofthe transaction at issue must be the addition of $125.00. If the valueis read correctly during the OCR process, the math will be consistentand the conversion and entry will be verified. If the OCR conversion isinaccurate, the math will not be consistent and an error may beidentified. In an embodiment, verified entries were identified usingunique screen identifiers, such as colored highlighting. Errors in anembodiment were identified in contrasting highlighting.

FIG. 3 represents a display in an embodiment using highlighting toindicate a series of verified transactions 300. Similarly, omitted orunverified OCR entries can be identified using visual identifiers, whichcan be quickly located on a display. FIG. 4 depicts transactions 400from the uploaded documents that could not be converted by the OCRengine for any one of a number of known reasons, such as smudges, papercreases, erasures, etc. The entries are either left without highlightingor are identified with contrasting highlighting Additionally, and/oralternatively, a flag may be associated with the data point identifyingand/or verifying a potential error or correctness of the associateddata. The flag may be used to cull out questionable entries.

In an embodiment, as seen in FIG. 4 when a transaction 400 cannot beread correctly via the OCR process, the system sets an alert that thetransaction in question should be reviewed. This may be done viahighlighting, or some other alert mechanism.

In an embodiment, after OCR conversion, the user may enter intoverification mode. During verification the system allows the user, viathe interface, to scrolls through all flagged potential discrepancies.In an embodiment, the verification tool allows the user tosimultaneously view the potentially-erroneous data as it was “read” bythe OCR engine while also viewing the image of the original documentfrom which the data was extracted. The transaction under review by theuser may be highlighted, and the read entry is displayed in an area ofthe display without obscuring the original image. As seen in FIG. 5entries 500 indicate entries that were converted correctly via OCR.Entry 501 indicates the entry under investigation, which has beenflagged by the system as potentially being incorrectly converted via OCRand area 502 displays the corresponding OCR data that was read from theoriginal document.

By cross-referencing the read data from area 502 with the actual entry501 itself on the statement, the user can easily identify if an errorexists. It will be understood that the identification of an entry as apotential error does not automatically imply an error occurred, merelythat the system is not one-hundred percent confident in the readfunction. As can be seen, in FIG. 5, the data in area 502 doescorrespond to the entry 501.

As will be understood, error correction provides for three primaryfunctions: 1) a review and revision the OCRed data including thetransaction date and/or the transaction amount; 2) the revision andentry of an entirely overlooked transaction into the database; and 3)the deletion of extraneous information that was incorrectly perceived asa transaction. In an embodiment, a lack of highlight may be used toalert the user that a transaction was overlooked entirely by the OCRengine. As seen in FIG. 6, a user can easily determine when this occursby viewing a page of transactional data with an eye on unshaded datasuch as area 601. As can be seen, in entry 601, the “2/26, State InsFund Wrkrs Comp, $160” indicates that the transaction was overlooked. Anoverlooked transaction causes the system to detect a perceived error inentry 602 because the reconciliation with account balances appears to beincorrect. Once the user inputs data into the system correcting theentry by adding the overlooked transaction to the database, the systemmay automatically re-reconcile.

In an embodiment, the system queries the user to review transaction 602.Although the transaction was actually read correctly, the running totalvalue does not reconcile with the running balance of the account due tothe overlooked transaction above. Because the system overlookedtransaction 601, transaction 602 is perceived as an error. Whenverifying the transaction value of −$6.25 for transaction 602, thesystem identifies that a mistake is present. To correct the problem theuser interfaces with the system via the user interface. The user may beprompted to select the appropriate behavior from area 600 such as NEWTXN (New Transaction) button 603 and enter the overlooked data as such.

FIG. 7 depicts the GUI display indicating that the new entry 701 hasbeen completed and that entry 702 is still in error. When the confirmbutton 703 is selected, the newly entered information will be stored tothe data store and the corresponding database will automatically update.The newly corrected data is dynamically updated to the database andcontinuously incorporates user-entered information to eliminate the needto inspect certain transactions that were incorrectly flagged for review(i.e., transactions that the system perceived were read incorrectly butin fact were read correctly), and to further focus in on actual errors.

Accordingly, as represented in FIG. 8, once the user enters and confirmsthe value of transactions 701 and 702, which were previouslyhighlighted, the system will automatically, in real time, display theentries as indicating that the values now reconcile with the runningtotal.

In an embodiment, the present system may identify certain data as atransaction, when in fact, such data from the input documents are nottransactions at all. These types of transaction entries, will also beflagged for user review. Because of the imprecise nature of OCR, thesystem must be able to identify when extraneous information, such as anaccount number, text on the footer of the page or text beneath a checkimage, is misidentified as a transaction. FIG. 9 represents a screenshot from a display of uploaded check images 900. As seen in FIG. 9, theinformation 901, below the check image has been incorrectly identifiedas part of the general list of transactions, when it is in fact anaccount number. The appropriate user behavior is to click the “Not Txn”button 903 in area 902 indicating that the highlighted information 901is not a transaction. Once the Not Txt button 903 has been clicked, thefalse transaction is removed from the database.

Since all transactional data must align with transaction balances, theerror correction process will yield one-hundred percent error free data.Even if a user makes a manual entry mistake, when trying to correct anerror in the statement, due to its real time updating, will notreconcile correctly and the system will generate an alert of thenumerical inconsistency in the same manner as if the entry had beenpulled via OCR.

There are many formats for presenting financial data. While presentationmay vary, the majority may be classified as running balance statements(balances are listed after every transaction), daily balance statements(balances are listed after every day in which transactional activityoccurs), and monthly balances statements (balances are listed at thebeginning and end of each monthly statement). The error correctionprocess utilized in the present disclosure is similar no matter whattype of statement is being reviewed; however, the more frequently thatbalances are listed, the quicker and more precisely the system can honein on errors. For example, in a statement within a daily balancetransaction register, if on February 12, there are three transactionsthat total $120 and the starting balance is $300 then the system expectsthat the beginning balance on February 13 will be $180. If however, thesystem converts the February 13 balance to $200, the system will detectthat an error occurred, but does not know which entry caused the erroror if the balance is in error. In an embodiment, all three transactionswill be identified for review as potential errors because the systemknows than an error has occurred.

Because of the nature of this statement, i.e., daily balance vs. runningbalance, the system cannot autonomously detect which of, or how many ofthe three transactions were misread. The system does automaticallydetect that an error occurred on February 12, but it cannot pinpointwhere exactly the error may be. If upon review, an entry is revised,e.g. one of the transactions is changed from −$50 to −$30, the userwould only need to review and correct that one specific transaction.This is because after the change and selecting confirm, the remainingtwo transactions will no longer be identified as errors as the systemautomatically, in real time, updates the series of stored displaytransactions. Accordingly, even though all three transactions wereinitially indicated as errors, only one correction is required for thesystem to move past all three prior identified transactions.

The same procedure applies on a larger scale for statements that onlylist beginning and ending monthly balances. If all of the transactionsare read correctly on the first pass, every transaction may behighlighted in green or indicative of a correct transaction, because thesystem will confirm the reconciliation. However, if even one erroroccurs during translation, all transaction for the indicated period willbe flagged as possible errors because the system cannot automaticallydetermine exactly where the error took place. If there is a singlemisread transaction on a statement that only lists beginning and endingmonthly balances, then on average, the user will only have to reviewhalf of that statement's transactions in order to find and rectify theone error.

As will therefore be appreciated, the system works most effectively onrunning balance statements, followed by daily balance statements andthen monthly balance statements. Once the error correction procedure iscompleted using verification mode, the result is a database ofone-hundred percent accurate transaction information.

FIG. 10 depicts a display on a user interface for investigation mode inaccordance with the present disclosure. In investigation mode, a usercan sort, search, filter, reorder and reorganize transaction data tolocate items of interest. In an embodiment, most transactions ofinterest can be located using fundamental search tools available ininvestigation mode. Typical, investigations may be conducted based ondate, dollar value, description, account source, deposits, transfers,withdrawals, debit card transaction checks, etc.

Sorting by value is especially useful because it allows the user toquickly and easily identify high-value transactions, repetitivetransactions or transactions of an exact amount. Similarly, a generalkeyword search allows for easy and rapid identification of parties paid,sources of income, cash transactions, debit card transactions andothers.

FIG. 11 depicts a display in investigation mode where the criteria hasbeen set to search for all deposits. As seen in FIG. 11, alltransactions are searched and as seen on the display, only the depositsare displayed for the user to review. In investigation mode, eachdisplayed transaction is dynamically linked back to the exact locationon the original uploaded document. Whenever a transaction in thedatabase is selected, the corresponding image of the appropriate pagefrom the uploaded document is displayed with the exact transactionhighlighted on the image. As seen in FIG. 12, image 1200 represents asnapshot of the entire uploaded statement allowing the user to focus onthe specific page where the transaction originated rather than thecomplete uploaded documents. Entry 1201 is the underlying datasupporting the transaction selected by the user while in investigationmode. Because the data entries are linked directly to the image, theuser can quickly contextualize any transaction to determine itssignificance to the investigation. Furthermore, by linking the entry inthe data with entry 1201, the user can again independently verify thatthe data presented is accurate.

In an embodiment of the present disclosure, the user is able toaggregate data from different sources and conduct searches on thecompiled data across the different various sources or accounts. That is,accounts from different sources, of different types, and of differentformats may all be uploaded and then treated as a single data store.

FIG. 13 depicts a screen that allows the user to select or filter whichaccounts the user wishes to search and to include or omit certainaccounts for each search. It will be appreciated by those skilled in theart, that different searches and filters can be combined to conduct veryprecise and efficient examinations. For example, in an embodiment, asearch can be used to locate deposits into a checking account or into asavings account held at different institutions or the same institution,that were made between date 1 and date 2 in any specific amount orgreater.

The present disclosure includes short cuts or hot buttons tool forconducting more advanced searches without having to format complex, butcommon searches. The advanced search feature provides users withdedicated tools for finding recurring transactions and transfers betweenaccounts. Unlike standard searches and sorts, the advanced searches maysimultaneously take into account all three major search parameters,e.g., date, description and dollar value. In an embodiment, the user cancreate and save their own advanced searches and allocate hot button keyson the display to user created buttons.

FIG. 14 depicts a user display in investigation mode, with advancedsearch buttons 1401 and 1402. As seen selecting button 1401 willidentify all detected transfers between accounts. Without any additionaluser action, once button 1401 is selected, the system of the presentdisclosure identifies and displays all transactions labeled with theword transfer and displays all exact dollar amount transfers betweenaccounts. Any transaction described with the word transfer or anabbreviation of the word transfer is automatically identified andhighlighted as indicated in entry 1403 as an identified transfer.Moreover, if the system detects an exact dollar amount match between onedebit 1404 transaction and one credit transaction 1405 which occurredwithin seven days of each other, the user is alerted via the probabletransfers list 1406.

Probable transfers list 1406 displays all debit transactions for whichan exact dollar amount match credit transaction exists within sevendays. Additionally, a user can initiate a deeper search for transfersthat accounts for the user's uncertainty in the dollar value and dateparameters of the search. One example of money movement that wouldtraditionally be difficult to track is, for example, if an individualwithdrew $13,000 from bank account A and then 21 days later deposited$12,300 into bank account B. Because the two transactions occurrednearly a month apart and are not of the same value, a user mightstruggle to realize that the two transactions are in fact associated.The present system's advanced search features allows the system toperform unique examinations by prompting the user to first select ananchor transaction, i.e., a transaction to serve as the basis of anadvanced search. If the user wishes to find a non-definite transfer, theuser can first identify a suspicious deposit or withdrawal as an anchortransaction 1501 and select find transfer matches 1502 as seen in FIG.15. As seen in FIG. 16, once the user selects find transfer matches1502, the user is then presented with a new menu 1600 with searchparameters to expand the search to locate potential matches for theanchor transaction 1501. In an embodiment, search menu 1600 includesparameters for a date range 1601 and a closeness in dollar value 1602(0%-25%). For example, if the anchor transaction 1501's value is $2,000and closeness in dollar value slider 1602 is set to 0%, the system willonly generate exact $2,000 matches. If the slider 1602 is set to 25%,the system will show potential matches ranging from $1,500 to $2,500.

In an embodiment, the present system also provides for a unique methodof locating recurring transactions. However, rather than only searchingfor exact matches, the system's recurring search allows a user to searchfor transactions with a spectrum input resolution rather than a seriesof binary filters. Recurring transactions are typically a series oftransactions that are related in terms of date, dollar amount and/ordescription. In many instances, recurring transactions occur on aregular basis, such as weekly, bi-weekly, or monthly. Recurringtransactions also typically have similar dollar amounts and/ordescriptions. Examples of recurring transactions include: sources ofincome, insurance payments, regular cash withdrawals, and receipt ofdividends. Since recurring transactions may only be loosely defined,classification of a transaction as a recurring transaction isalgorithmically indeterminate and requires user discretion. Accordingly,in order to identify recurring transactions, the user first identifiesan anchor transaction in a recurring series that will serve as the basisfor the search.

As seen in FIG. 17, transaction 1701 is identified as the anchortransaction which will serve as the basis for searching out all relatedrecurring transactions. The date (not seen), the description 1702 anddollar amount 1703 of this anchor transaction 1701 will be comparedagainst all of the other transactions in the database to locate the mostlikely potential matches. When the user selects find recurrences 1704the user will be prompted with menu 1800 which allows the user to conveyto the system how important each parameter is. The user can selectcriteria regarding amount, description date and frequency which will inturn generate a results list of the most probable matches for the userto review and confirm or reject as matches.

FIG. 19 depicts the results list from a recurrence search. In anembodiment, the criteria depicted on menu 1800 are not exact because ifthe user defines the exact parameters of a search using date and dollarvalue, it is possible that the transaction for which the user issearching will be missed because some parameters may be just beyond theuser-defined values. For example, a user searching for a certaintransaction between $1000 and $1150 during July and August of 2014 maymiss a transaction credited to the account on September 1. However,using looser or fuzzy search criteria, the definitions used during therecurring search feature of the system benefits from an indefiniteapproach whereby all potential matches are sorted by estimatedprobability of match. In that manner, a user does not have to decide ona hard cutoff when looking for transactions, and instead may searchconfidently even in the face of uncertainty. Another benefit of thesystem's recurring search feature, is that it not only allows foruncertainty in the search process, but it accounts for uncertainty byincorporating it into the search process. When a user selects an anchortransaction 1701 as the basis of the user's recurring search, the useris presented with the parameters seen in menu 1800.

Typically, basic searches are a logical union or intersection of binaryfilters, an approach that suffers from loss of resolution. That is,searches and especially recurring searches, use inputs that moreaccurately represent and quantify a spectrum. For example, suppose thatthe recurring transactions that are being searched for are socialsecurity payments. In that case, the dollar amount of the transactionsin question (first slider in menu 1800) are very consistent.Additionally, the description (second slider of menu 1800) of thepayments as they appear on different bank statements are fairlyconsistent, and there is loose consistency of the monthly recurrence(third and fourth sliders) because, although Social Security is paid outmonthly, it is not paid on the same exact day every single month. Unlikea traditional search, the recurring search in the present disclosuretakes into account varying degrees of certainty in the three parameters(date, description and dollar value), and uses that uncertainty to theuser's benefit in quantifying potential matches.

As seen in FIG. 19, the result of the search is a sorted list of alltransactions in the database, wherein the sorting order, i.e., theposition of any given transaction on the list, depends on the estimatedprobability that the given transaction is a match, meaning that it isclassified as part of the same recurring series as the anchortransaction 1701.

Once a user selects all the appropriate matches 1901, selects save andnames the series (in this case Social Security), the transactions 1901which were selected will be permanently highlighted and labeled. As seenin FIG. 20, the previously identified transaction 1901 are allhighlighted and labeled 2001 as being part of the same recurringtransaction.

The present system may run on a server, a series of servers, a networkor a series of network. The system may be run on a cloud based platformor a local network. In an embodiment, the system runs on a virtualprivate network (VPN), such as Amazon VPN with commercial levelinfrastructural security of the uploaded content. More specifically,because the system provides for the input of financial and other privateinformation, security of that information is paramount to systemoperation. Computer security of the present system comprises mainlypreventive measures, e.g. firewalls. A firewall can be defined as a wayof filtering network data between a host or a network and anothernetwork, such as the Internet, and can be implemented as softwarerunning on the machine, to provide real time filtering and blocking.Another implementation may be a physical firewall which consists of aseparate machine filtering network traffic.

FIG. 21 depicts the main components of the system in an embodiment andcomprises a VPN 2100, a virtual private cloud (VPC) 2101, a gateway2102, a two way gateway 2103 and a corporate gateway 2104 coupled to adata center 2105. VPN 2100 may be a private network or a commercialnetwork such as Amazon Web Services. VPN 2100 extends a private networkacross a public network, such as the Internet. The VPN 2100 allowsusers, using a browser, to send and receive data over shared or publicnetworks as if their computing devices were directly connected to theprivate network. In the manner, the users benefit from thefunctionality, security and management policies of the private network.VPC 2101 may be part of a commercial cloud computing service such asAmazon, that provides users with a virtual private cloud space, byallocating and provisioning a logically isolated section of a cloudplatform. Users are able to access the VPN over an IP based network. VPN2102 is a gateway that allows secure communication into the privatecloud between VPC 2101 and the VPN 2100. VPN connection 2103 enablesusers to connect between data center 2105 via gateway 2104 and the VPN2100 and communicate to the VPC 2102. This connection allows the user tointeract with the instances within VPC 2101 as if the users were withinthe existing network. Gateway 2104 establishes the link between thecustomer datacenter 2015 and VPN 2100. Customer datacenter 2105 is theuser data center as the user may be accessing the system via web browseror other network portals.

FIG. 22, depicts the program components of system 2200. System 2200 maycomprise at least 3 major components, with each component comprisingmultiple sub-components. As will be appreciated by those skilled in theart, more or less of these components may be used and more or less maybe combined into other components. The components of system 2200 may beimplemented on computers, servers, workstations, or any other hardwareor software based platform.

Workflow application interface (API) service 2101 may comprise one ormore scalable, non-blocking web server and web application frameworksuch as a Tornado web server. Servers 2100 may serve web contents withnon-blocking network I/O to maximize system throughput. Message broker2102 may be a message-oriented middleware application that implementsadvanced messaging protocol utilizing hardware or software of both.Message broker 2102 may be RabbitMQ or similar type service, such as aRedis message broker for passing the transient content such as processbatch status, etc. between the workflow API 2201 and the OCR engine2203. Storage servers 2204 may hold all user documents in a secure andredundant form, to ensure both security and availability. OCR Engine2203 may comprises a series of servers running an asynchronous taskqueue/job queue based on distributed message passing. The servers withinOCR engine 2203 focus on real-time operation, but supports scheduling aswell. The servers may comprise single or multiple core platforms and maybe implemented on a single machine, or multiple machines. In anembodiment, a Celery engine provided the required parallel processing ofthe scanned input document. The execution units of the engine 2203,called tasks, may be executed concurrently on a single or more workerservers using multiprocessing, events. The tasks can executeasynchronously (in the background) or synchronously (wait until ready).At the core of technical stack are OCR and transaction recognitionprocedures.

As indicated above, the system of the present disclosure allows for theuploading of scanned or electronic statements in a variety of formats.

The heart of the system relies on OCR engine 2203. Optical characterrecognition (OCR) is the mechanical or electronic conversion of imagesof typed, handwritten or printed text into machine-encoded text. It isoften considered a field of research in pattern recognition, artificialintelligence and computer vision. It is widely used as a form of dataentry from printed paper data records. It is a common method ofdigitizing printed texts so that it can be electronically edited,searched, stored more compactly, displayed on-line, and used in machineprocesses such as machine translation, text-to-speech, key data and textmining.

Very often to enhance the functionality, OCR “pre-processes” images toimprove the chances of successful recognition. Some OCR pre-processingtechniques that may be used in the present system include, but are notlimited to de-skewing, such as in the case of the document not beingaligned properly when scanned; de-speckling, which is the removal ofpositive and negative spots and smoothing the edges; binarization, whichincludes converting an image from color or greyscale to pureblack-and-white. The task of binarization may be performed as a simpleway of separating the text (or any other desired image component) fromthe background. Additionally, the binarization task is often necessarysince most recognition algorithms work only on binary images.

Some other preprocessing technique that may be used, is line removal,which cleans up non-glyph boxes and lines; layout analysis or zoning,which identifies columns, paragraphs, captions, or other features asdistinct blocks. This is especially important when processingmulti-column layouts and tables. Line and word detection helpsestablishes a baseline for word and character shapes and may separatewords if necessary. Character isolation or segmentation is apreprocessing process for per-character OCR. That is, multiplecharacters that are connected due to image artifacts must be separatedprior to recognition and single characters that are broken into multiplepieces due to artifacts must be reconnected.

As will be understood, OCR accuracy can be increased based on the amountof preprocessing performed. It may further be enhanced if the expectedoutput is constrained by a lexicon, i.e., a list of words that areexpected and allowed to occur in a document. Such a lexicon may be forexample, all the words in the English language, or a more technicallexicon for a specific field. For example, tesseract uses its dictionaryto influence the character segmentation step, for improved accuracy.More sophisticated OCR systems such as the one embodied in the presentdisclosure can preserve the original layout of the page and produce, forexample, an annotated PDF that includes both the original image of thepage and a searchable textual representation.

Near-neighbor analysis is another technique that may be used to enhanceaccuracy. Near-neighbor analysis can make use of co-occurrencefrequencies to correct errors, by noting that certain words are oftenseen together. Knowledge of the grammar of the language being scannedalso helps determine if a word is likely to be a verb or a noun, forexample, allowing greater accuracy. As will be understood, beyond anapplication-specific lexicon, better performance can be had by takinginto account business rules, standard expression, or rich informationcontained in color images. Such a strategy that may be employed in thepresent system is known as “application-oriented OCR” or “customizedOCR”. Such systems may be used to OCR of license plates, business cards,invoices, screenshots, ID cards, driver licenses, and automobilemanufacturing.

Nevertheless, despite technological advancement, OCR accuracy can stillbe confused by some or all of the factors and preprocessing stepsdiscussed above. In the field of financial statement, for example, thepresent system can greatly reduce the number of OCR misreads byaccumulating a knowledge base of templates used by financialinstitutions in the period of past 30 years, for example to aid with theOCRing of a specific document.

Most OCR algorithms produce the read as tree-like structure, forexample, starting at a root of the tree, the root node represent thepage, which can have multiple child nodes to represent paragraphs, whichagain may have multiple child nodes to represent a line of text, then aword, and finally, a character. It is typical that at every level,locational information is associated with the read in the form of abounding box on the page. Additionally, in some OCR algorithms, aconfidence level can be produced along with the text and locationalinformation.

General purpose OCR algorithms try to process every character, image,lines, etc. on any given page. On the other hand, when considering afinancial statement, a large portion of any financial statement may beignored if the purpose of OCR read is to locate and gather transactioninformation. It is therefore advantageous to have knowledge of therelationship of key information (such as the name of the bank, statementmonth and year, account number, begin and ending balance, etc.) and thelocation where the system expects to look for and find this information.It is therefore critical to have a mechanism to programmatically proofread program results. That is, the program's behavior is most oftendeterministic. Meaning, that if it performs the same procedure multipletimes is should not yield more optimistic results. However in thepresent system, it is common to engage in a trial and error approach tomaximize OCR accuracy, that involves performing a multi-step imagepre-processing, subsequently followed by a transaction detection, whichoften results in multiple execution paths that have overlappingsub-paths. This multi-step approach is important to the unique executionoptimization employed in the present system.

The system handles both processing and storage of the large volume ofsuch uploaded documents. This poses two technical challenges. First,document processing time and second, document storage, transactiondetail storage, and searches.

In an embodiment, to estimate resources required, the first step is tocollect information about the conversion exercise and the processingoptions that are available. Typical criteria may includedocument-related data; number of documents; average number of pages perdocument; image type (Black & White, Grayscale, Full Color); type ofinput file (TIFF, PDF . . . ); document resolution (200 dpi, 300 dpi . .. ); typical document page size (US Letter, A4, A3 . . . ); typicaldocument text density; available processing options; auto-rotate;de-skew; line removal; de-speckle; PDF/A; compression, etc.

Second, the system needs to estimate the complexity and generate acomplexity rating to assign to the operation. Complexity rating may beapproximated based on assigning a rating for the following criteria:

TABLE 1 Low Medium High Very High TIFF Image Source200 DPI PDF FileColor TIFF or Color MRC US Letter/A4 Source300+ PDF Compression MediumText Density DPIHigh Text ImagesLarge Bitonal/Black & WhiteDensityGrayscale Format ImagesDeskew Images Auto-rotate Line RemovalBitonal Compression PDF/A

Next, to properly allocate and design the system, the estimating PagesPer CPU Core Hour (PPCCH) must be approximated. OCR and compressionprocessing may be highly CPU-intensive so in an embodiment, ahigh-performance server with Intel® i5 processor or better was used.Based on the complexity scale, a PPCCH value may be generated.

TABLE 2 Complexity PPCCH Low 3000 Medium 1500 High 900 Very High 600

Once the PPCCH value is calculated, number of CPU cores required can becomputed. While the theoretical maximum number of CPU Cores that areavailable is based on the number of servers * number of CPU cores/serverthere are a number of other factors that can reduce this absolutenumber.

First, if the system is largely dedicated for OCR use, it is recommendedto leave at least 1 CPU Core available for non-OCR system use. Inaddition if other applications and services are running it is prudent tobe conservative about the number of cores available.

Second, by default OCR jobs will only use one CPU core. It is possibleto increase this to a maximum of 10 cores if multi-core usage isavailable. Another option may be to use, a threads setting which isintended for use with documents that have a large number of pages (200+)since it works by splitting document into 2 or 4 chunks and processingeach in parallel. Another alternate approach is to configure multipleconcurrent parallel jobs in OCR—i.e. have 2 or 4 jobs that can run inparallel.

Once the number of cores is identified, the estimated time required torun the conversion can be computed. The number of hours required is

$\frac{P}{\left( {{PPCCH}*C} \right)},$

where P=Number of Pages, C=Number of usage CPU Cores, and PPCCH=Pagesper CPU Core Hour. In an example, a medium complexity conversion job of500,000 pages using two CPU cores would require 500,000/(1500*2)=167hours to complete. Accordingly, in order to process the 167 hours ofwork in 8 hour requires both computing in parallel (accomplished by theOCR engine 2203 using the Celery workers) and advanced technique such asusing GPU (Graphic Computing Unit). As will be understood, to reduceprocessing time the system can rely on additional cores. Accordingly,processing may be accomplished in a number of hours or minutes.Specifically, the present system may process even a large job comprisingseveral thousands of pages in 1 minute, 5 minutes, 15 minutes, 30minutes, etc.

In the second phase of the process, the requirements for documentstorage, and transaction detail storage and inquiry needs to beidentified. More specifically, an image file for single pagedocumentation typically occupies at least kilobytes in size.Accordingly, using a typical example of document size and complexity, itis clear that the storage requirement for one year's worth of data ofuploaded documents may reach into the terabytes in size. Additionally,OCR software often requires image pre-processing that can easily doubleor triple the storage requirement. In the present system, imagepre-processing must be accomplished in the most effective ways possiblein control and minimize the volume of work.

In typical financial statements, the transaction section of thefinancial statement is often formatted such that natural languageprocessing capability of OCR does not necessary enhance read accuracy,but instead only adds program inefficiency. Instead, in the presentsystem, the OCR engine 2203 is able to use transaction data afterfinishing the OCR “reading”, thereby using the detection results as afeedback mechanism to provide hints of how well OCR did. For example, acompletely upside down page image will likely result in poor OCR readwhich can be captured by its confidence level. However, a page thatcontains multiple check images will likely result in low confidencelevel as well, which makes it an unreliable predictor. In these cases,transaction detection results can be used to influence the OCR engine2203 to perform a variety of pre-processing steps, including page imagerotation, for example

A simplistic implementation of utilizing transaction detection relies onregular expressions to identify transactions in a line by line fashion,with each line abiding to certain rules or patterns. In reality,however, a transaction can be spread across multiple lines, with partsof each line containing information belonging to a differenttransactions, etc. In those cases, segregating the page into informationblocks becomes a very convenient way to enhance the OCR performance.

The present system uses knowledge about the layouts of the financialstatement to help improve either read accuracy or read performance orboth. Absent that, the system's ability to scan forward and backward andperform cross reference of the data is an absolute requirement. Theuniqueness and advantage of the present system starts with a generic OCRapproach to ensure good coverage on font, language character set, color,etc. next, the system takes a practical approach to narrow down OCR readon the identified blocks that are of interest to the specificapplication, i.e., the transactions. This is determined by searching thestatement and matching it to a template in a database that learns andgets more comprehensive over time.

Another aspect of the present system allows for the system to identifyand highlight an identified bounding box for any financial transactiondetected by the system. As seen in FIG. 23, individual financialtransactions are identified by the bounding boxes 2301 a and 2301 b andidentified as individual transactions or objects 2302 a and 2302 b. Thepresent system utilizes the bounding box to identify the geographiclocation of the information on the scanned document and generates aspatial index linking the X and Y location of the information on thedocument to the transaction that has been extracted. The informationwithin a bounding box is then treated as a single transaction.

In another aspect of the present disclosure, Directed Acyclic Graph(DAG) may be used to improve performance. A DAG is a directed graph withno directed cycles. That is, it is formed by a collection of verticesand directed edges, with each edge connecting one vertex to another,such that there is no way to start at some vertex v and follow asequence of edges that eventually loops back to v again. DAGs may beused to model many different kinds of information. The reachabilityrelation in a DAG forms a partial order, and any finite partial ordermay be represented by a DAG using reachability. A collection of tasksthat must be ordered into a sequence, subject to constraints thatcertain tasks must be performed earlier than others, may be representedas a DAG with a vertex for each task and an edge for each constraint;algorithms for topological ordering may be used to generate a validsequence. Additionally, DAGs may be used as a space-efficientrepresentation of a collection of sequences with overlappingsubsequences. DAGs are also used to represent systems of events orpotential events and the causal relationships between them. DAGs mayalso be used to model processes in which data flows in a consistentdirection through a network of processors, or states of a repository ina version-control system.

An application of directed acyclic graphs arises in the conciserepresentation of a set of sequences as paths in a graph. For example,the directed acyclic word graph is a data structure in computer scienceformed by a directed acyclic graph with a single source and with edgeslabeled by letters or symbols; the paths from the source to the sinks inthis graph represent a set of strings, such as English words. Any set ofsequences can be represented as paths in a tree, by forming a tree nodefor every prefix of a sequence and making the parent of one of thesenodes represent the sequence with one fewer element; the tree formed inthis way for a set of strings is called a trie. A directed acyclic wordgraph saves space over a trie by allowing paths to diverge and rejoin,so that a set of words with the same possible suffixes can berepresented by a single tree node.

The present system relies on DAG to model the image (pre)processing, OCRand transaction detection components. For example, if there are 3 imagepre-processing steps available, e.g., X:Image binarization; Y:Gaussianelimination; and Z:image rotation (90 degree), depending on the qualityof the original document, the present system may choose to perform 1, 2or all 3 steps. As seen in FIG. 24 there are overlapping sub-path on theexecution path.

As used in the present disclosure, by developing and utilizing a properDAG implementation, an intermediate result can be cached that results inthe elimination of duplicate computations. For example, if theorientation of a document page requires 270 degree rotation(preprocessing step Z), the present system will perform the binarizationstep (X) only once, cache the processed image, rotate the process imageby 90 degrees (Z), cached the result, then perform another 2 rotation(Z)(Z) of 90 degrees are performed, one on top of the other in step wisemanner. In this way, the processing speed is increased because thebinarization step needs only to be performed once and not on all threerotations. This improvement increases accuracy, reduces processing powerand improves overall conversion speed. The DAG in the enables the systemto identify the effectiveness of the preprocessing and allows the systemto determine if alternative preprocessing steps are required. In thisway the amount of steps and time required to perform effectivepreprocessing is reduced because the number of potential computations ofpreprocessing techniques is reduced and not duplicative.

The following exemplary pseudo-code illustrates how this mechanism worksin the present system, where binarization is evaluated only once.

from mdf import MDFContext, evalnode, varnode image = varnode( )@evalnode def binarization( ):  return ostu-method(image ( )) @evalnodedef rotate( ):  return image_rotate_by_degree(binarization( ), 90) ctx =MDFContext( ) # set image to be processed ctx[image] =image(File(′/pdf/image/image_001.png′)) # calculate stepwise processedimage by modifying and layering the context scenarios = [90, 180, 270]for s in scenarios:  shifted_ctx = ctx.shift({binarization:rotate(s-90)})  processed_image = shifted_ctx[rotate]

In an embodiment, the Mdf decorators (such as @evalnode, @varnode) wereimplemented in Cython, thereby giving maximum performance.

Additionally, to further improve processing speed of the OCR andtransaction detection performance, the system may run the OCR componentsof the OCR engine 2203 on GPU. By running on GPU, it allows GPUcomputation by building on top of a GPU enabled image library such asopencv, and a high-order logic implementation such as pydatalog

GPUs are Single Instruction, Multiple Data (SIMD) machines. They are aclass of parallel computers they consist of many processing elementsthat all run the same program but on distinct data items. This kernel,can be quite complex and includes control statements such as if andwhile statements. However, the kernel may be synchronized by hardware,i.e., where each instruction within the kernel is executed across allprocessing elements running the kernel. Thus, if a kernel has to comparestrings, processing elements that compare longer strings will takelonger and the other processing elements will wait for them.Accordingly, the scheduling of GPU in the present disclosure may work asfollows.

A thread in the host platform (e.g., a multi-core) first copies the datato be processed from host memory to GPU memory, and then invokes GPUthreads to run the kernel to process the data. Each GPU thread has anunique id which is used by each thread to identify what part of the dataset it will process. When all GPU threads finish their work, the GPUsignals the host thread which will copy the results back from GPU memoryto host memory and schedule new work. GPU memory is typically organizedhierarchically as shown in FIG. 25 Each GPU thread 2501 has its ownper-thread local memory 2502. Threads are grouped into blocks 2503, eachblock having a memory 2504 shared by all threads in the block. Finally,thread blocks are grouped into a single grid 2505 to execute akernel—different grids can be used to run different kernels. All gridsshare the global memory 2506. The global memory 2506 is the GPU “mainmemory”. All data transfers between the host (CPU) and the GPU are madethrough reading and writing global memory.

OpenCV GPU or other open source code modules are a set of classes andfunctions to utilize GPU computational capabilities. They may beimplemented using a NVIDIA* CUDA* Runtime API and may support onlyNVIDIA GPUs. The OpenCV GPU module for example, includes utilityfunctions, low-level vision primitives, and high-level algorithms.

The standard way to provide feedback to OCR which may work well for onecategory of documents may perform poorly on other classes of document,such as financial statements from one institution vs. another. Thechallenge in processing bank statement, is that training an algorithm to“learn” how to best process financial statements varies largely fromformats and contents between statements.

Accordingly, the present system solves this program by providing an easyway to use user verification tools to provide feedback. For example, itis known to have a separate execution of OCR and transaction detectionper page in order to cut down overall execution time, by a parallelmechanism. However this approach poses a challenge in transactionaccuracy. Some crucial information related to the financialtransactions, such as statement year/month, account number, ownerinformation, etc. may only appear on certain pages. To solve thisproblem, the present system uses a consolidation routine to fill in thecritical pieces of information that may not be available on each page ofa statement.

In the present system, by using a datalog, such as Pydatalog the systemsupports high-order logic in functional forms. Pydatalog is used toperform a machine learning algorithm that improves transaction detectioncomponent accuracy over a large training data set. The training data maycome from a relational data store that contains both raw OCR output andpost-process financial transaction information with a high degree ofaccuracy. FIG. 26 illustrates the work flow of an exemplary machinelearning engine in abstract term.

The following pseudo-code illustrates how such a algorithm may beimplemented:

class Transaction(pyDatalog.Mixin): # --> A transaction (date,description, amount, account, other_acct)  def__init__(self, date,description, amount, account, other_acct):   super(Transaction,self).__init__( )  # calls the initialization method of the Mixin class  self.date = date   self.amount = amount   self.description=description    ...  @pyDatalog.program( )  # indicates that thefollowing method contains pyDatalog clauses  def Transfer(from, to,amount):   # A transfer is a withdraw from account_from and a deposit inaccount_to   # the following is a logical predicate vs a funtional one.  Transaction.Transfer[date, from, to, amount] <= from & to &Transaction(date, _, amount,from, to) & Transaction(date, _, amount, to,from) # An example how a transfer is detected. acct1 = ′001′ acct2 =′002′ acct3 = ′003′ trans1 = Transaction (′20150108′, ′transfer fromchecking acct into saving acct′, 6800, acct1, acct2) trans2 =Transaction (′20150108′, ′transfer from checking acct into saving acct′,-6800, acct2, acct1) trans3 = Transaction (′20150208′, ′XXX′, 800,acct1, acct3) trans4 = Transaction (′20150208′, ′YYY′, -800, acct3,acct1) # query the sum of all transfers originated from acct1 ? sum(X,X=pydatalog.ask(′tranfer(acct1, _, X)′)

Accordingly, by simply defining these simple rules, the deduction enginecan work its way out to improve transaction accuracy such as identifyingaccount transfers by taking advantage of a deduction engine. In asystem, Pydatalog decorators were implemented by manipulating python AST(abstract syntax tree).

The utilization of high-order logic in processing bank statement isunique in the industry. High-order logic enables the reasoning engine todetect relationships between seemingly unrelated transaction High-orderlogic is a form of predicate logic that is distinguished from lower orfirst-order logic by additional quantifiers and a stronger semantics.Higher-order logics with standard semantics are more expressive, but themodel-theoretic properties are less well-behaved than those offirst-order logic. Higher-order logic is the union of first-, second-,third-, . . . , nth-order logic; i.e., higher-order logic admitsquantification over sets that are nested arbitrarily deeply.

FIG. 27 depicts a general computer architecture, such as that foundwithin a server or mobile device on which the present teaching can beimplemented and has a functional block diagram illustration of ahardware platform which includes user interface elements. The computermay be a general purpose computer or a special purpose computer. Thiscomputer 2700 can be used to implement any components of the system andmethod for processing and manipulating transaction data using dataprocessing as described herein. For example, the OCR engine 2203 can beimplemented on a computer such as computer 2700, via its hardware,software program, firmware, or a combination thereof. Although only onesuch computer is shown, for convenience, the computer functions relatingto the systems and methods for processing and manipulating transactiondata using data processing may be implemented in a distributed fashionon a number of similar platforms, to distribute the processing load.

The computer 2700, for example, includes COM ports 2750 connected to andfrom a network connected thereto to facilitate data communications. Thecomputer 2700 also includes a central processing unit (CPU) 2720, in theform of one or more processors, for executing program instructions. Theexemplary computer platform includes an internal communication bus 2710,program storage and data storage of different forms, e.g., disk 2770,read only memory (ROM) 2730, or random access memory (RAM) 2740, forvarious data files to be processed and/or communicated by the computer,as well as possibly program instructions to be executed by the CPU. Thecomputer 2700 also includes an I/O component 2760, supportinginput/output flows between the computer and other components thereinsuch as user interface elements 2780. The computer 2700 may also receiveprogramming and data via network communications.

In one or more embodiments, a proprietary OCR system is combined with aspecialized back-office program, which achieves reconciliation throughhuman editing and validation of OCR results. The back-office program (BOTool) provides certain features and functionality as further describedbelow. Prior to the application of OCR, the system may perform apreprocessing on the data. In one or more embodiments, the preprocessingincludes at least one of de-skewing, de-specking, binarization, lineremoval, layout analysis, zoning, line detection, word detection,character isolation, segmentation, applying a lexicon, near neighboranalysis, grammatical analysis, applying Benfords law, or any number ofcombinations of the foregoing.

According to one embodiment, as shown with reference to FIG. 28, thesystem implements a process 2800 that begins after an end-user uploads abank or credit card statement as a PDF file at step 2802. In step 2804,the document is reviewed using the BO Tool to identify the documenttype. The use of the BO Tool may be implemented automatically through anartificial intelligence system, such as a machine vision system, or itcan be implemented by a user interacting with the BO Tool. Becausecustomers may mistakenly upload documents other than bank or credit cardstatements, the system must determine, in step 2804, whether or not anuploaded document is a bank or credit card statement. If it is not, thenthe file is rejected, and in step 2806 the system sends the customer anotification that the document is not able to be processed, and nofurther processing is performed on the document. The review of theuploaded file or document may be accomplished automatically through anartificial intelligence system, such as a machine vision system, or maybe implemented by a user undertaking a manual review of the uploadedfile or document, or a combination of both. If the document isconfirmed, in step 2804, to be a bank or credit card statement, thesystem or reviewer will confirm, in step 2808, whether it is displayedin portrait orientation. If not, in step 2810, the system canautomatically, or the user can manually, such as, for example, through amouse click, rotate selected pages or all pages. A thumbnail view can beused to quickly locate improperly oriented pages and correct theirorientation.

After identification of the document type and proper orientation of thedocument, in step 2812, the system identifies various subportions of thedocument and categorizes those subportions. The identification and/orcategorization can be accomplished automatically through an artificialintelligence system, such as a machine vision system, or can beimplemented by a user interacting with the BO Tool, or a combination ofboth. The categorization can be based, for example, on assigning anidentifier or a code to each subportion of the document. For example, ifthe document is a bank statement, the inputted values for beginningbalance and ending balance are identified. These values may be verifiedin a later process. In order to provide the balance data for this laterverification process, an image, or a “snippet,” of the beginning balanceand ending balance must be taken from the image of the entire page.Thus, as shown in FIG. 29, either the system will automatically, or auser will manually, use the BO Tool to draw a bounding box around thebeginning and ending balances, effectively creating a snippet, as shownin FIG. 30, that will be saved and later reviewed and verified byautomatically by the system, or manually by another user, if necessary.Upon submission of this task, the system will assign the appropriateAccount Identifier to each transaction identified in the document. Atstep 2814 the document is automatically divided into subdocuments basedon its content, and the position of the segmentations that separatesubdivisions may be edited through the BO Tool, which can be implementedusing the identifiers or codes assigned to the various subportions ofthe document in step 2812.

In one or more embodiments, the primary method of segmentation is bybeginning account balance and ending account balance. An uploadeddocument may contain many months of bank statements, which can belogically split where the beginning and ending balances appear on thestatement. If the statement does not contain beginning and endingmonthly account balances, the document may be split, automatically or bya user manually interacting with the system, into the smallestsubsections which contain all necessary information for that subsectionto be further parsed independently. To enact the page splitting, eitherthe system will automatically, or the user interacting with the systemwill manually, enter the page ranges for each section in the “Statementsection” text box, as depicted in FIG. 31. Alternatively, using a manualapproach, the user will press the “Mark as start” and “Mark as end”buttons to denote the sections. Pages without any relevant bankinginformation may be omitted. The omission eliminates the need for laterreview of pages that have no relevant information.

Following the subdivision of the document, in step 2816, the document isanalyzed using optical character recognition (OCR). In step 2818, theoutput of the OCR is fed into the BO Tool for data verification. The BOTool breaks down the data verification process into microtasks that canbe simultaneously completed automatically through an artificialintelligence system, such as a machine vision system, or can beimplemented by one or more users interacting with the BO Tool, or acombination of both. When implemented by users interacting with the BOTool, multiple users may be utilized with each user being assigned adifferent microtask in order to maximize throughput. The microtasks alsoallow for labor specialization and minimization of labor downtime.

In one or more embodiments, as shown in FIG. 32, implemented throughuser interaction, a hierarchy of bank statement data is used to optimizethe parallelization of user tasks in order to achieve the fastestoverall turnaround of bank data to clients as well as to minimizebottlenecks in labor utilization. In step 3200 a document is inspectedby a user. After initial inspection by the user, to ensure documentswill be properly processed, in step 3202, the document is divided intogroups of pages. In step 3204, sets of metadata for each group of pagesare automatically generated algorithmically, and in step 3206, themetadata is then validated by user review. In step 3208, each pagewithin a group of pages is then automatically analyzed algorithmicallyto identify predefined transactions and/or the nature of thetransaction, each predefined transaction is annotated, and then eachpage is displayed individually with the annotatedalgorithmically-recognized transactions. The annotations may include,for example, highlighting of the algorithmically-recognizedtransactions, or placement of bounding boxes around thealgorithmically-recognized transactions, or different color shading ofthe algorithmically-recognized transactions. The bounding boxcoordinates that were determined in the OCR process prior to thedocument entering the BO Tool, are used to highlight each individualtransaction. The BO Tool displays an image of the transactions withtheir bounding boxes shaded, as shown in FIG. 33.

The annotations may also be used to denote or represent different,algorithmically-inferred accounts. In one or more embodiments, a userreview is implemented for purposes of quality control, which includesreview of each page and confirmation that all transactions are properlybounded and shaded. As part of the user review, the user will review thepolarity (+/−) sign of the transaction, as automatically assigned by thealgorithm. For example, the polarity sign of the transaction may bedenoted with an icon in the bounding box. Any errors in theautomatically generated identifier may be corrected by the user throughinteraction with the system.

In one or more embodiments, using the BO Tool, the user is able toquickly scan through the transactions and rectifies any errors in thealgorithmically-inferred polarity (+/−) of the transaction. The BO Tooldisplays the sign of each transaction to the left of the bounding box,and the user views the transactions, judges whether the displayed signsare correct, and in the case that they are not, rectifies any errorsthrough a radio editing button.

In step 3210, each relevant page is sliced into transaction images, forexample, from within bounding boxes. In step 3212, a Moderation Flowprocess begins. In Moderation Flow, the system prompts the user toverify transactional and account data from snippets until reconciliationis achieved. After reconciliation, in step 3214, the edited transactionand account data is compiled into Final SME Review, Final SME Reviewincludes a page-by-page human review of all data from each bankdocument. After Final SME Review, in step 3216, the verified bankstatement data is returned to the customers through either a webapplication or an API.

In one or more embodiments, a review process, as shown in FIG. 34, isimplemented after either of steps 2814, 2816 or 2818 of FIG. 28. Thisprocess begins after the system has divided the document into thevarious sections, as described above. In step 3400, a section of thedocument is presented for review. The review of the section of thedocument may be accomplished automatically through an artificialintelligence system, such as a machine vision system, or may beimplemented by a user undertaking a manual review of the section of thedocument, or a combination of both. In one or more embodiments, in step3402, the account holder's Personal Identifiable Information (PII) andthe account metadata, such as, for example, information that applies tothe entire account within a statement, as shown in FIG. 35, istranscribed or edited automatically through an artificial intelligencesystem, such as a machine vision system, or may be implemented by a userundertaking a manual review of the section of the document, or acombination of both.

In one or more embodiments, in step 3404, the accuracy of the accountnumber, bank name, and account type (e.g., checking, savings, etc.) maybe reviewed and confirmed automatically by the system or manually by auser. The combination of the account number, bank name and account typeis colloquially called the Account Identifier. The back-end OCR andtransaction detection algorithms will attempt to automatically identifythis data and auto-populate the corresponding fields, as shown in FIG.35. This automatic identification can be subjected to quality control byan automated review, for example, through an artificial intelligencesystem, such as a machine vision system, or by a manual review by a userfor the purpose of comparing the fields to the data as it appears on thedocument, such as, for example, the bank statement, and to rectify anyerrors that are generated by the automated system. In the event that theautomated system cannot resolve the bank name and account type, thecorresponding field will be left blank. In the automated review, thesystem, and in the manual review, the user, will enter the correct datainto the blank fields. Alternatively, or additionally, a dropdown menuwill present the most probable autocompletion possibilities, from whichthe user may choose.

When the document represents a bank statement, because some statementsare consolidated, i.e., they contain information from multiple accounts,such as a joint checking and savings statement, metadata for multipleaccounts within a section may need to be created. In step 3406, thismetadata creation may be accomplished automatically through anartificial intelligence system, such as a machine vision system, or maybe implemented by a user undertaking manual data input into the sectionof the document, or a combination of both. When the system,automatically, or a user, manually, creates a second account within thesection, some of the metadata from the first account is automaticallytranscribed to the second account. The automatic transcription savestime because much of the metadata from the first account, such asaccount holder Personally Identifiable Information and bank name, alsoapplies to the second account. The system, automatically, or the user,manually, will change the second account's metadata to reflect whateverdiscrepancies exist between the second account's data and the back end'sdata.

In one or more embodiments, in step 3408, verification of the accountnumbers is accomplished automatically through an artificial intelligencesystem, such as a machine vision system, or is implemented manually by auser. This task is only necessary when there are two or more accounts onthe same page, in a section of bank statements. The account of eachtransaction is resolved from the list of different accounts created asdescribed above, and is ultimately determined automatically through anartificial intelligence system, such as a machine vision system, or byhuman judgment. The bounding boxes are shaded in different colors byaccount, which serve as a visual identifier for the automated or manualverifier, allowing the verifier to quickly determine if the transactionsare assigned correctly.

In step 3410, missing transactions are identified, located and added,and extraneous non-transactions are deleted. Missing transactions arethose which were not recognized by the back end algorithm and thus notshaded on the BO Tool with a bounding box. Non-transactions are boundingboxes that do not envelope a transaction. Non-transactions are deletedautomatically through an artificial intelligence system, such as amachine vision system, or manually by the simply by clicking on the “x”to the right of the bounding box. Missing transactions are located bythe automated system by adding, or manually by a user by dragging, a boxaround the missing transaction. When a missing transaction is locatedand added, the automated system or the user will assign it to an accountand assign polarity. Actions such as deletions and assignments ofaccount and polarity can be performed automatically by the system onmultiple transaction at once, or manually on multiple transaction atonce by holding a key down while selecting them.

In one or more embodiments, the transaction date, description and amountof the highlighted transaction are viewable for the purpose ofidentification, but are not editable in step 3410. Accordingly, when amissing transaction is added, the fields for date, description andamount will be blank. This information will undergo OCR in a latermachine step.

In step 3412, after all of the transactions have been split into‘snippets’, the transactions, i.e, the image within the bounding boxesfrom transactions are displayed to the user. Below each image are fieldsfor the corresponding date and description. The fields are populatedwith the respective OCR outputs, and the system and/or the user mustrectify any errors in the fields, as shown in FIG. 36.

At this point, the system has enough information to determine whetherthe transactional data from the statement section is correct, or whetheran error has occurred. In step 3414, the determination is made usingreconciliation, which is based on the premise that the arithmeticdifference between the ending balance and the beginning balance of astatement must equal the sum of the amounts of all transactions.

If the statement does not reconcile, meaning at least one error hasoccurred, in step 3416—reconciliation is conducted for each accountwithin a statement section. Step 3416 is undertaken only in the casethat one or more accounts in a statement section do not reconcile.

A failure for an account to reconcile means that either the data fromone or more transaction amounts is incorrect, and/or the data of theaccount balances is incorrect. It is more efficient to check for errorsin the account balances before checking for errors in the transactions.Thus, the program displays the snippets of beginning and ending balancesand their corresponding values, as shown in FIG. 37. The system,automatically through an artificial intelligence system, such as amachine vision system, or manually by a user undertaking manual datainput into the section of the document, or a combination of both, checksfor and rectifies errors in the beginning and ending account balances.Once the monthly balances have been validated, the system will attemptto reconcile all accounts. If reconciliation is achieved for all periodswithin all accounts, then Moderation Flow will end and the last step,Final SME Review, will begin.

If reconciliation is not achieved, then in step 3416, the user isdisplayed the transaction snippets for each account period that did notreconcile. Below each image is a field for the dollar amount. The fieldis populated with the OCR output, and the system, automatically throughan artificial intelligence system, such as a machine vision system, ormanually by a user undertaking manual review, any errors in the fieldare rectified. After each transaction amount is validated, the programwill attempt to reconcile the accounts. If the accounts reconcile, instep 3418, Moderation Flow will end. If the accounts fail to reconcilethe above process is repeated a second time. In step 3420, if theaccount balances still do not reconcile, in step 3422, the documentsection is sent to Final SME Review for further inspection. The statusof the reconciliation will be displayed to the user in the last step,Final SME Review.

In step 3422, the Final SME Review, the transactional and other bankdata is culled from all microtasks and presented to the user. The datapresented to the user during Final SME Review, as shown in FIG. 38,presents a visual summary of the bank data for the purposes ofreviewing, editing and finalizing before it is tabulated and returned tothe client, either through a web application or API.

Some symbols representing transactional information are simultaneouslyvisible to the reviewer for all transactions. Other pieces of data areonly visible for one transaction at a time, and are presented to theuser when that transaction is clicked on.

The information that is always visible to the reviewer includes thepolarity (+/−sign), the reconciliation status (red or green color) andseveral others, including:

Green “M”: If the balances tie then there will be a green “M” next tothe amount. A user does not have to verify amounts in these cases.

Red “M”: If the balances do not tie the “M” will be in red and willindicate to the groomer that he/she needs to verify each amountindividually. In the event, there is a mistake in one of the amounts orbalances, the groomer must make the update and hit “save.”

Red Exclamation Point: This means that the date and/or description andamount was marked as “unreadable” and did not go through the entireprocess of verification. Transactions with this marker need to have thedates/descriptions and amounts verified.

A red “R” on the transaction signifies that the running balances do nottie. In these situations a user must look at each transaction with the“R” and ensure that the amount is correct. If it is not, it needs to beedited.

Duplicate Detection: Any transaction with the “dupe” symbol means thatthere is a high percentage chance that there are duplicate transactionscontained in the statement (see, e.g., FIG. 39). For example, sometimeschecks appear in the main section of transactions on a statement andalso in a separate section that contains check images. Duplicates mustbe identified and removed so that the tabulated list of transactionsthat is returned to the client does not contain multiple entries for thesame transaction. Thus, the user will click on the transaction, and thecorresponding transactions that are most likely to be a duplicate willbe shaded more prominently. The user will examine the differentpossibilities and determine whether the transaction is a duplicate. Ifthe pair is in fact a duplicate, the user must select the “x” to theright of the bounding box (and to the immediate left of the “dupe”symbol) to delete it from the database.

As shown in FIG. 40, the data that is visible for a single transactionis presented in the panel on the right hand side. It includesTransaction Type, Account, Statement Period, Transaction Date,Description and Amount.

As shown in FIG. 41, other pieces of data apply to a group oftransactions that are tied to a particular account, though only datafrom one transaction at a time is shown: Statement Period, BeginningBalance (if applicable), Ending Balance (if applicable), Begin Date, EndDate, Account Number, Bank i.e. financial institution, and account Type.

As shown in FIG. 42, there is depicted a process for analyzing adocument according to an embodiment of the present disclosure. First, adocument is input into the system, where it is split into differentpages, and then prepared as separate pages. Next, a document type andsections are identified, following which the document is logged, and thedocument status is updated in a database. From the document logging, thedocument is split into different sections, which are then logged. Thesystem collects accounts, statement period dates and balances. Theautocomplete data stores are updated with this information and thesections are logged. Next, the different sections are split intoseparate pages. Next, transactions are fetched from the data store andthey are mapped to the different pages, and a default account is set forthe transactions on each page. Next, the transactions are identifiedwith sign logic and are mapped to the associated account, after whichthe pages are again logged. Next, the pages are sliced in to transactionimages and then transactions are verified and extracted, and a statemanagement period is set for each transaction. The dates andmodifications are then logged. Next, pages are joined to sections toaccurately reflect the transactions and accounts, where the processmoves to Moderation Flow. In Moderation Flow, monthly balances areverified. Transactions are split for amount verification, amounts areprepared and verified, amounts are logged and joined to the transactionfor a particular corresponding group and then verified. Additionally,images are prepared for balances verification, beginning and endbalances are verified and corrected balances are saved and modificationsare logged, and the monthly balances are then verified. Upon completionof Moderation Flow, the information is sent for Final SME Review. InFinal SME Review, modifications are logged, the results are written tothe data store, and the data is updated in the database.

The system offers guidance as to which transactions are most likely toneed editing. The system also attempts to infer which transactions aremore likely to be erroneous among a set of transactions that failreconciliation. The system computes the probability based on variousfactors, such as, for example, OCR confidence on a character and wordbasis as well as the type of statement and type of transaction. Forexample, in one or more embodiments, those transactions that are morelikely to be erroneous are bounded in a darker shade of red compared totransactions that are less likely to be erroneous.

After reconciliation has been achieved, or when the automated systemdetermines or the user conducting the Final SME Review believes that thedata is sufficiently accurate, the results will be sent to the client.The bank statement data that was validated in the BO tool will be storedin a database. The client can then request the data using the API orinteract with the data in a web application, such as for example, aPerfectAudit web app.

Hence, aspects processing and manipulating transaction data using OCRand data transactions, as outlined above, may be embodied inprogramming. Program aspects of the technology may be thought of as“products” or “articles of manufacture” typically in the form ofexecutable code and/or associated data that is carried on or embodied ina type of machine readable medium. Tangible non-transitory “storage”type media include any or all of the memory or other storage for thecomputers, processors or the like, or associated modules thereof, suchas various semiconductor memories, tape drives, disk drives and thelike, which may provide storage at any time for the softwareprogramming.

All or portions of the software may at times be communicated through anetwork such as the Internet or various other telecommunicationnetworks. Such communications, for example, may enable loading of thesoftware from one computer or processor into another, for example, froma management server or host computer of the search engine operator orother explanation generation service provider into the hardwareplatform(s) of a computing environment or other system implementing acomputing environment or similar functionalities in connection withgenerating explanations based on user inquiries. Thus, another type ofmedia that may bear the software elements includes optical, electricaland electromagnetic waves, such as used across physical interfacesbetween local devices, through wired and optical landline networks andover various air-links. The physical elements that carry such waves,such as wired or wireless links, optical links or the like, also may beconsidered as media bearing the software. As used herein, unlessrestricted to tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

Hence, a machine readable medium may take many forms, including but notlimited to, a tangible storage medium, a carrier wave medium or physicaltransmission medium. Non-volatile storage media include, for example,optical or magnetic disks, such as any of the storage devices in anycomputer(s) or the like, which may be used to implement the system orany of its components as shown in the drawings. Volatile storage mediainclude dynamic memory, such as a main memory of such a computerplatform. Tangible transmission media include coaxial cables; copperwire and fiber optics, including the wires that form a bus within acomputer system. Carrier-wave transmission media can take the form ofelectric or electromagnetic signals, or acoustic or light waves such asthose generated during radio frequency (RF) and infrared (IR) datacommunications. Common forms of computer-readable media thereforeinclude for example: a floppy disk, a flexible disk, hard disk, magnetictape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any otheroptical medium, punch cards paper tape, any other physical storagemedium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM,any other memory chip or cartridge, a carrier wave transporting data orinstructions, cables or links transporting such a carrier wave, or anyother medium from which a computer can read programming code and/ordata. Many of these forms of computer readable media may be involved incarrying one or more sequences of one or more instructions to aprocessor for execution.

In an embodiment, the described systems and methods may be implementedin the device operating system and applied to all geo-locationinformation. In an embodiment, the described systems and methods areimplemented using a preloaded application. In an embodiment, the systemsand methods described herein are implemented using middleware betweenthe operating system and an application. In an embodiment, the systemsand methods are implemented in a cloud-based environment, where thedevice and application are in constant communication with a cloud-basedserver. In such an embodiment, the system's requests for measurementsare all sent to the device and all measurements are sent back to thecloud. In an embodiment, the described methods may be implemented in astand alone device without internet connectivity.

As will be appreciated by one skilled in the art, aspects of thisdisclosure can be embodied as a system, method or computer programproduct. Accordingly, aspects of the present disclosure can take theform of an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or asembodiments combining software and hardware aspects that can allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the disclosure can take the form of a computerprogram product embodied in one or more computer readable medium(s)having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) can beutilized. The computer readable medium can be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium can be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific example (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium can be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium can include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal can takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium can be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium can be transmittedusing any appropriate medium, including but not limited to wireless,wire line, optical fiber cable, radiofrequency (RF), etc., or anysuitable combination of the foregoing.

Computer program code for carrying out operations for aspects of thepresent disclosure can be written in any combination of one or moreprogramming language, including an object oriented programming language,such as Java, Smalltalk, C++ or the like and conventional proceduralprogramming language, such as the “C” programming language or similarprogramming languages. The program code can execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer can be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection can be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present disclosure has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the form disclosed. Many modifications and variations will beapparent to those of ordinary skill in the art without departing fromthe scope and spirit of the disclosure. The embodiments were chosen anddescribed in order to best explain the principles of the disclosure andthe practical application, and to enable others of ordinary skill in theart to understand the disclosure for various embodiments with variousmodifications as are suited to the particular use contemplated.

The diagrams depicted herein are illustrative. There can be manyvariations to the diagram or the steps (or operations) described thereinwithout departing from the spirit of the disclosure. For instance, thesteps can be performed in a differing order or steps can be added,deleted or modified. All of these variations are considered a part ofthe disclosure.

The terminology used herein can imply direct or indirect, full orpartial, temporary or permanent, action or inaction. For example, whenan element is referred to as being “on,” “connected” or “coupled” toanother element, then the element can be directly on, connected orcoupled to the other element and/or intervening elements can be present,including indirect and/or direct variants. In contrast, when an elementis referred to as being “directly connected” or “directly coupled” toanother element, there are no intervening elements present.

Although the terms first, second, etc. can be used herein to describevarious elements, components, regions, layers and/or sections, theseelements, components, regions, layers and/or sections should notnecessarily be limited by such terms. These terms are used todistinguish one element, component, region, layer or section fromanother element, component, region, layer or section. Thus, a firstelement, component, region, layer, or section discussed below could betermed a second element, component, region, layer, or section withoutdeparting from the teachings of the present disclosure.

The terminology used herein is for describing particular exampleembodiments and is not intended to be necessarily limiting of thepresent disclosure. As used herein, the singular forms “a,” “an” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. The terms “comprises,” “includes”and/or “comprising,” “including” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence and/oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof.

Example embodiments of the present disclosure are described herein withreference to illustrations of idealized embodiments (and intermediatestructures) of the present disclosure. As such, variations from theshapes of the illustrations as a result, for example, of manufacturingtechniques and/or tolerances, are to be expected.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this disclosure belongs. Theterms, such as those defined in commonly used dictionaries, should beinterpreted as having a meaning that is consistent with their meaning inthe context of the relevant art and should not be interpreted in anidealized and/or overly formal sense unless expressly so defined herein.

Furthermore, relative terms such as “below,” “lower,” “above,” and“upper” can be used herein to describe one element's relationship toanother element as illustrated in the accompanying drawings. Suchrelative terms are intended to encompass different orientations ofillustrated technologies in addition to the orientation depicted in theaccompanying drawings. For example, if a device in the accompanyingdrawings were turned over, then the elements described as being on the“lower” side of other elements would then be oriented on “upper” sidesof the other elements. Similarly, if the device in one of the figureswere turned over, elements described as “below” or “beneath” otherelements would then be oriented “above” the other elements. Therefore,the example terms “below” and “lower” can encompass both an orientationof above and below.

As used herein, the term “about” and/or “substantially” refers to a+/−10% variation from the nominal value/term. Such variation is alwaysincluded in any given value/term provided herein, whether or not suchvariation is specifically referred thereto.

If any disclosures are incorporated herein by reference and suchdisclosures conflict in part and/or in whole with the presentdisclosure, then to the extent of conflict, and/or broader disclosure,and/or broader definition of terms, the present disclosure controls. Ifsuch disclosures conflict in part and/or in whole with one another, thento the extent of conflict, the later-dated disclosure controls.

It will be understood that those skilled in the art, both now and in thefuture, can make various improvements and enhancements which fall withinthe scope of the claims which follow.

What is claimed is:
 1. A method comprising: accessing, by a processor,an image depicting a set of data including a set of dates, a set oftransaction descriptors, a set of transaction amounts, and a set ofbalance amounts arranged to form a set of transaction rows and a set ofcategory columns such that each row of the set of transaction rows (a)includes a date sourced from the set of dates, a transaction descriptorsourced from the set of transaction descriptors, a transaction amountsourced from the set of transaction amounts, and a balance amountsourced from the set of balance amounts and (b) is arranged according tothe set of category columns; recognizing, by the processor, a firstsubset of the set of data in the image based on a first set of criteriaand a second subset of the set of data in the image based on a secondset of criteria, wherein the first subset includes a first transactionrow of the set of transaction rows arranged according to the set ofcategory columns, wherein the second subset includes a secondtransaction row of the set of transaction rows arranged according to theset of category columns, wherein the first row is distinct from thesecond row; causing, by the processor, each of the first subset depictedin the image and the second subset depicted in the image to besimultaneously presented via a display; causing, by the processor, afirst bounding box to be presented over the first transaction row in afirst visual manner via the display and a second bounding box to bepresented over the second transaction row in a second visual manner viathe display while the first bounding box is presented over the firsttransaction row, wherein the first visual manner is distinct from thesecond visual manner; receiving, by the processor, a user input from auser input device while the first bounding box is presented over thefirst transaction row via the display and the second bounding box ispresented over the second transaction row via the display; and causing,by the processor, the second bounding box to change from being presentedin the second visual manner to the first visual manner via the displaybased on the user input.
 2. The method of claim 1, wherein the firstbounding box has a first perimeter, wherein the second bounding box hasa second perimeter, wherein the first visual manner is distinct from thesecond visual manner based on the first perimeter being colored distinctfrom the second perimeter.
 3. The method of claim 1, wherein the firstbounding box has a first perimeter, wherein the second bounding box hasa second perimeter, wherein the first visual manner is distinct from thesecond visual manner based on the first perimeter being shaped distinctfrom the second perimeter.
 4. The method of claim 1, wherein the firstbounding box has a first perimeter, wherein the second bounding box hasa second perimeter, wherein the first visual manner is distinct from thesecond visual manner based on the first perimeter being sized distinctfrom the second perimeter.
 5. The method of claim 1, wherein the firstbounding box encloses a first area presented over the first transactionrow, wherein the second bounding box encloses a second area presentedover the second transaction row, wherein the first visual manner isdistinct from the second visual manner based on the first area beingcolored distinct from the second area.
 6. The method of claim 1, whereinthe user input is a first user input, and further comprising:recognizing, by the processor, a third subset of the set of data in theimage based on a third set of criteria distinct from each of the firstset of criteria and the second set of criteria, wherein the third subsetincludes a third transaction row of the set of transaction rows arrangedaccording to the set of category columns, wherein the third row isdistinct from each of the first row and the second row, causing, by theprocessor, the third subset depicted in the image to be presented viathe display while each of the first subset depicted in the image ispresented via the display and the second subset depicted in the image ispresented via the display; causing, by the processor, a third boundingbox to be presented over the third transaction row in a third visualmanner via the display while the first bounding box is presented overthe first transaction row in the first visual manner and the secondbounding box is presented over the second transaction row in the secondvisual manner, wherein the third visual manner is distinct from each ofthe first visual manner and the second visual manner; receiving, by theprocessor, a second user input from the user input device while thefirst bounding box is presented over the first transaction row via thedisplay, the second bounding box is presented over the secondtransaction row via the display, and the third bounding box is presentedover the third transaction row via the display; and causing, by theprocessor, the third bounding box to change from the third visual mannerto the first visual manner via the display based on the second userinput.
 7. The method of claim 1, further comprising: receiving, by theprocessor, a file including a set of pages; and forming, by theprocessor, a set of images from the set of pages, wherein the set ofimages includes the image before each of the first subset and the secondsubset is identified.
 8. The method of claim 1, further comprising:causing, by the processor, a graphical user interface (GUI) to bepresented via the display, wherein the GUI presents each of the firstsubset in the image and the second subset in the image while the firstbounding box is presented over the first transaction row and the secondbounding box is presented over the second transaction row, wherein theGUI presents a previous user input element and a next user input elementeach programmed for a transaction-based navigation based on the set oftransaction rows when respectively activated.
 9. The method of claim 8,wherein the GUI is programmed to present a new transaction user inputelement programmed to initiate an entry of data for a new transactionthat was overlooked in the set of data when the new transaction userinput element is activated.
 10. The method of claim 1, wherein the firstset of criteria is distinct from the second set of criteria.
 11. Asystem comprising: a server programmed to: access a file selected by theclient; form a set of images from the file, wherein the set of imagesincludes an image depicting a set of data including a set of dates, aset of transaction descriptors, a set of transaction amounts, and a setof balance amounts arranged to form a set of transaction rows and a setof category columns such that each row of the set of transaction rows(a) includes a date sourced from the set of dates, a transactiondescriptor sourced from the set of transaction descriptors, atransaction amount sourced from the set of transaction amounts, and abalance amount sourced from the set of balance amounts and (b) isarranged according to the set of category columns; recognize a firstsubset of the set of data in the image based on a first set of criteriaand a second subset of the set of data in the image based on a secondset of criteria, wherein the first subset includes a first transactionrow of the set of transaction rows arranged according to the set ofcategory columns, wherein the second subset includes a secondtransaction row of the set of transaction rows arranged according to theset of category columns, wherein the first row is distinct from thesecond row; cause each of the first subset depicted in the image and thesecond subset depicted in the image to be simultaneously presented onthe client; cause a first bounding box to be presented over the firsttransaction row in a first visual manner on the client and a secondbounding box to be presented over the second transaction row in a secondvisual manner on the client while the first bounding box is presentedover the first transaction row, wherein the first visual manner isdistinct from the second visual manner; receive a user input from theclient while the first bounding box is presented over the firsttransaction row on the client and the second bounding box is presentedover the second transaction row on the client; and cause the secondbounding box to change from being presented in the second visual mannerto the first visual manner on the client based on the user input. 12.The system of claim 11, wherein the first bounding box has a firstperimeter, wherein the second bounding box has a second perimeter,wherein the first visual manner is distinct from the second visualmanner based on the first perimeter being colored distinct from thesecond perimeter.
 13. The system of claim 11, wherein the first boundingbox has a first perimeter, wherein the second bounding box has a secondperimeter, wherein the first visual manner is distinct from the secondvisual manner based on the first perimeter being shaped distinct fromthe second perimeter.
 14. The system of claim 11, wherein the firstbounding box has a first perimeter, wherein the second bounding box hasa second perimeter, wherein the first visual manner is distinct from thesecond visual manner based on the first perimeter being sized distinctfrom the second perimeter.
 15. The system of claim 11, wherein the firstbounding box encloses a first area presented over the first transactionrow, wherein the second bounding box encloses a second area presentedover the second transaction row, wherein the first visual manner isdistinct from the second visual manner based on the first area beingcolored distinct from the second area.
 16. The system of claim 11,wherein the user input is a first user input, wherein the server isfurther programmed to: recognize a third subset of the set of data inthe image based on a third set of criteria distinct from each of thefirst set of criteria and the second set of criteria, wherein the thirdsubset includes a third transaction row of the set of transaction rowsarranged according to the set of category columns, wherein the third rowis distinct from each of the first row and the second row; cause thethird subset depicted in the image to be presented on the client whileeach of the first subset depicted in the image is presented on theclient and the second subset depicted in the image is presented on theclient; cause a third bounding box to be presented over the thirdtransaction row in a third visual manner on the client while the firstbounding box is presented over the first transaction row in the firstvisual manner and the second bounding box is presented over the secondtransaction row in the second visual manner, wherein the third visualmanner is distinct from each of the first visual manner and the secondvisual manner; receive a second user input from the client while thefirst bounding box is presented over the first transaction row on theclient, the second bounding box is presented over the second transactionrow on the client, and the third bounding box is presented over thethird transaction row on the client; and cause the third bounding box tochange from the third visual manner to the first visual manner on theclient based on the second user input.
 17. The system of claim 11,wherein the server is further programmed to: access an optical characterrecognition (OCR) engine; recognize the first subset of the set of datain the image based on the first set of criteria after the OCR engineprocesses the image and the second subset of the set of data in theimage based on the second set of criteria after the OCR engine processesthe image.
 18. The system of claim 11, wherein the server is furtherprogrammed to: present a previous user input element and a next userinput element on the client, wherein the previous user input element andthe next user input element are respectively programmed for atransaction-based navigation based on the set of transaction rows whenrespectively activated on the client.
 19. The system of claim 18,wherein the server is programmed to present a new transaction user inputelement on the client, wherein the new transaction user input element isprogrammed to initiate an entry of data for a new transaction that wasoverlooked in the set of data when the new transaction user inputelement is activated on the client.
 20. The system of claim 11, whereinthe server is programmed to: form a first copy of the first subset afterthe first subset has been recognized; form a second copy of the secondsubset after the second subset has been recognized; present aninvestigation mode on the client such that each of the first copy andthe second copy is presented in a tabular format.
 21. The system ofclaim 11, wherein the first set of criteria is distinct from the secondset of criteria
 22. A method for auditing transactions comprising;receiving, over a communications network, a set of image basedtransactions; storing the set of image based transaction in a centraldata store; associating the set of image based transactions with aunique identifier associated with a user; identifying a transactionformat from the set of image based transactions, utilizing a processorto apply a preprocessing to the set of image based transactions based onthe identifying; processing the preprocessed image based transactions,via the processor, into a series of text based transactions, whereineach image based transaction has a related text based transaction andeach text based transaction has a plurality of data representing thetransaction; storing, the plurality of data for each text basedtransaction in a data store; associating a quality identifier with eachtext based transaction wherein the quality identifier is based on theprobability that the text based transaction was processed correctly;applying an identifier to a text based transaction based on the qualityidentifier; transmitting, over the network, to a display the text basedtransaction and the associated identifier.